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MULTIMEDIA DOCUMENT ANNOTATION 



FIELD OF THE INVENTION 

The present invention relates generally to field of multimedia creation 
and presentation. More specifically, the present invention is directed to adding 
multimedia annotations to paper or electronic documents. 

BACKGROUND 

Multimedia is a term used to describe the ability to combine different 
kinds of information storage and /or communication media, such as sounds, 
video, text, music, animations, charts, maps, etc., into colorful, interactive 
presentations, business applications, games, etc. Examples of information 
storage media include books, phonograph records, audio and videotapes, 
microfilm, and magnetic and optical disks. 

Audio and video clips require enormous amounts of storage space, and 
for this reason, until recently, programs could not use any but the most 
rudimentary animations and sounds. The enormous storage capacity of current 
storage devices such as the compact disc read only memory (CD-ROM) changes 
all that. When using simultaneous clips from several different media, the user's 
senses of sight, sound, and touch are merged into an astonishingly real 
experience. 

Faster computers and rapid proliferation of multimedia programs will 
probably forever change the way people get information. The computer's 
ability to instantly retrieve a tiny piece of information from the midst of a huge 
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mass of data has always been one of its most important uses. Since video and 
audio clips can be stored alongside text on a single CD-ROM disc, a whole new 
way of exploring a subject is possible. By using hyperlinks, materials can be 
presented to people so that they can peruse it in a typically human manner, by 
5 association. 

Although there are technologies today that allow for the implementation 
of multimedia, current implementations do not address the incorporation of 
multimedia with the traditional form of office communication, the paper 
document. Since most communications within an office today are still made 
10 with the traditional paper documents such as memos and notes, it would be 
advantageous to be able to add multimedia to the paper documents and enable 
the office user to convey the sound and emotion with the paper document. 
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SUMMARY OF THE INVENTION 



A method and apparatus for adding multimedia annotations to paper 
documents is disclosed. In one embodiment, a multimedia annotation is created 
for a paper document. The multimedia annotation is combined with the paper 
document to form a multimedia document. 
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BRffiF DESCRIPTION OF THE DRAWINGS 



The present invention is illustrated by way of example in the following 
drawings in which like references indicate similar elements. The following 
drawings disclose various embodiments of the present invention for purposes of 
illustration only and are not intended to limit the scope of the invention. 

Figure 1 illustrates a flow diagram of one embodiment of a process of 
annotating. 

Figure 2 depicts one embodiment of a multi-fimction system for 
annotating documents with the multimedia information. 

Figure 3 illustrates an exemplary document with a multimedia 
annotation represented with a bar code. 

Figure 4 illustrates an exemplary email message and a Uniform Resource 
Locator (URL) that provides a link to the multimedia content. 

Figure 5 illustrates an exemplary embodiment of a system that provides 
for adding multimedia annotation with emotional content to a document. 

Figure 6 illustrates one embodiment of a computer system. 

Figure 7 illustrates one embodiment of a computer-readable medium 
containing various sets of instructions, code sequences, configuration 
information, and other data used by a computer or other processing device. 
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DETAILED DESCRIPTION 



A method and apparatus for adding multimedia annotations to paper 
documents is disclosed. In the following description, for purposes of 
5 explanation, specific nomenclature is set forth to provide a thorough 

understanding of the present invention. However, it will be apparent to one 
skilled in the art that these specific details are not required in order to practice 
the present invention. For example, the sending of the multimedia document by 
email is described with reference to an Internet. However, the same techniques 

10 can easily be applied to other types of network. 

Some portions of the detailed descriptions that follow are presented in 
terms of algorithms and symbolic representations of operations on data bits 
within a computer memory. These algorithmic descriptions and representations 
are the means used by those skilled in the data processing arts to most 

15 effectively convey the substance of their work to others skilled in the art. An 
algorithm is here, and generally, conceived to be a self-consistent sequence of 
steps leading to a desired result. The steps are those requiring physical 
manipulations of physical quantities. Usually, though not necessarily, these 
quantities take the form of electrical or magnetic signals capable of being stored, 

20 transferred, combined, compared, and otherwise manipulated. It has proven 
convenient at times, principally for reasons of common usage, to refer to these 
signals as bits, values, elements, symbols, characters, terms, numbers, or the like. 

It should be borne in mind, however, that all of these and similar terms 
are to be associated with the appropriate physical quantities and are merely 

25 convenient labels applied to these quantities. Unless specifically stated 
otherwise as apparent from the following discussion, it is appreciated that 
throughout the description, discussions utilizing terms such as "processing" or 
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"computing" or "calculating" or "determining" or "displaying" or the like, refer to 
the action and processes of a computer system, or similar electronic computing 
device, that manipulates and transforms data represented as physical 
(electronic) quantities within the computer system's registers and memories into 
5 other data similarly represented as physical quantities within the computer 

system memories or registers or other such information storage, transmission or 
display devices. 

The present invention also relates to apparatus for performing the 
operations herein. This apparatus may be specially constructed for the required 

10 purposes, or it may comprise a general-purpose computer selectively activated 
or reconfigured by a computer program stored in the computer. Such a 
computer program may be stored in a computer readable storage medium, such 
as, but is not limited to, any type of disk including floppy disks, optical disks, 
CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random 

15 access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or 
any type of media suitable for storing electronic instructions, and each coupled 
to a computer system bus. 

The algorithms and displays presented herein are not inherently related 
to any particular computer or other apparatus. Various general-purpose 

20 systems may be used with programs in accordance with the teachings herein, or 
it may prove convenient to construct more specialized apparatus to perform the 
required method steps. The required structure for a variety of these systems 
will appear from the description below. In addition, the present invention is not 
described with reference to any particular programming language. It will be 

25 appreciated that a variety of programming languages may be used to implement 
the teachings of the invention as described herein. 
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Overview 

In one embodiment, a method and apparatus for creating and adding 
multimedia annotation to paper or electronic documents to create a multimedia 
document is described* 
5 Figure 1 is a flow diagram of one embodiment of a process for adding 

multimedia contents to a document. Referring to Figure 1, at block 105, the user 
creates a document. In one embodiment, the document is a paper document 
(e.g., hand written document, a document created through the use of a 
typewriter, a printed copy, etc.). The user then generates a multimedia 

10 annotation to accompany the document, as shown in block 110. In one 

embodiment, the multimedia annotation can be an audio sound, a video clip, a 
combination of both the audio sound and the video clip, etc. The multimedia 
annotation can be generated at the same time the document is created or at a 
different time. At block 115, the annotation is associated with the document to 

15 create a multimedia document. In one embodiment, the annotation is placed on 
an area of the document. The aimotation may be in different forms, such as, for 
example, a bar code containing an audio message or a URL indicating a link to a 
video clip. 

In one embodiment, the user specifies a recipient to receive the 
20 multimedia document. The recipient may be specified by the user entering the 
recipient's address such as, for example, an Internet email address. The email 
message is generated and sent to the recipient, as shown at block 120. 
Alternatively, the user may choose to specify a recipient for the document at 
another time. At block 125, the document and the annotation are saved in a 
25 storage area. The document and the annotation can be saved together or they 
can be saved in separate storage areas. 



Ricoh 



-7- 



07445 .PI 14 



Figure 2 shows one embodiment of a multi-function system. In one 
embodiment, the multi-function system 202 can be a photocopier equipped with 
a display console 204 and a keypad 205. The multi-function system 202 may 
include a microphone 210, a video camera 215, and a disk drive 220 to accept 
5 input generated from a different system. Furthermore, in one embodiment, the 
multi-function system 202 includes hardware for digitizing the audio and /or 
video multimedia inputs. The formats of the audio sound and the video clip 
may include, for example, MPS audio (MPEG audio layer 3) and MPEG video (a 
digital video compression standards from the Motion Picture Experts Group). 

10 In one embodiment, the user generates the multimedia annotation while 

using the multi-function system 202 to copy a paper document. While copying 
the paper document, the multi-function system 202 captures an image of the 
paper document and saves it on the server machine 225 automatically. In 
another embodiment, the user may generate the document in electronic form 

15 and then used the electronic form of the document with the multi-function 
system 202. 

The server machine 225 is coupled to the multi-function system 202 
through a network such as, for example, a local area network. The server 
machine 225 is used to store the multimedia documents so that it can be 
20 retrieved by the user or by the person to whom the user sends the multimedia 
document. In one embodiment, the console 204 is used to specify the 
recipient(s) of the multimedia document. The automatic saving feature of the 
system allows users to save document images while requiring no decision on the 
part of the user. 

25 In one embodiment, the microphone 210 is automatically activated when 

the user selects a copy function on the multi-function system 202. The user 
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generates the multimedia annotation by speaking to the microphone 210 on the 
multi-function system 202. 

In another embodiment, the user may prepare the multimedia annotation 
prior to using the multi-function system 202. For example, the user generates 
5 the multimedia armotation on a workstation, records it on a portable storage 
device such as, for example, a floppy disk, and physically carries the portable 
storage device to the multi-function system 202. Upon inserting the portable 
storage device in the drive 220 on the copier, the user is presented with an 
interface on the console 204. In one embodiment, the interface lists all the 

10 available multimedia armotations stored on the portable storage device and 
allows the user to review and select them. The user may interact with the 
interface through a user controllable device coupled to the multi-function 
system 202, such as, for example, a mouse. Additionally, a keyboard 225 may be 
provided to allow the user to perform any editing functions, such as, for 

1 5 example, specifying or changing the names of the recipients for the documents. 
The interface of the multi-fimction system 202 may provide a mechanism for the 
user to associate a selected multimedia annotation with the document. For 
example, a drag-and-drop function to indicate which multimedia armotation 
should be applied to the document being copied. 

20 In one embodiment, the user may use the keyboard 225 in conjunction 

with the console 204 to indicate information about the recipient of the 
document, such as, for example, names, email address, phone number, fax 
number, etc. 

The server machine 225 may be configured to allow access only after 
25 authentication. For example, only the recipient who possesses a valid user 
identification and/ or password may access the corresponding multimedia 
documents stored on the server machine 225. In one embodiment, to provide 
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additional security, the multimedia document may also be encrypted. For 
example, the multi-function system 202 may utilize a digital key belonging to 
the recipient to encrypt the document. In this manner, only those individuals 
having the recipient digital key may decrypt the code and recover the original 
5 document. 

In one embodiment, the multimedia document generated by the multi- 
function system 202 can be a physical document such as, for example, a paper, a 
transparency, etc. This may be in addition to the document image that was 
automatically stored. For example, a paper document generated by the copier 

10 function of the multi-function system 202 may contain a plain text 

representation for a Uniform Resource Locator (URL) that indicates the network 
address where the multimedia annotation is stored. An individual may 
distribute this paper version of the multimedia document to the recipients. 
Additionally, an individual may decide to notify the recipients by email that 

15 provides them with the URL for the multimedia document. In another 
embodiment, the URL where the multimedia document is stored can be 
represented by a bar code generated by the multi-function system 202 and 
printed on the document. 

Figure 3 illustrates an exemplary embodiment of a document with the 

20 multimedia annotation represented as a bar code. In this case, the document 305 
can be a paper document and the complete multimedia annotation can be 
represented with bar code 310 printed on the face of the document. The 
multimedia annotation can be retrieved by decoding the bar code 310. In 
altemative embodiments, the multimedia annotation may be emailed using 

25 digital links as described in U.S. Patent number 5,337,362 issued to Gormish et 
al. on August 9, 1994. 



Ricoh 



-10- 



07445.P114 



Figure 4A illustrates an exemplary embodiment of an email message 
containing a URL that represents the link to the multimedia document. The 
recipients of multimedia documents can be notified by the email message 405 
that includes a URL 410 for the multimedia document. The email message may 
5 be generated by the user at the time the multi-function system is activated (e.g., 
selecting the copy function). The recipients then can access the multimedia 
document (document and multimedia annotation) by selecting the URL 410. 

Figure 4B illustrates an exemplary embodiment of a multimedia 
document. In one embodiment, upon selecting the URL, the image of the 

10 multimedia document 412 is presented to the recipient and, at the same time, a 
multimedia player is invoked and run at the recipient's workstation. The 
multimedia player plays the multimedia annotation that is part of the document. 
Alternatively, the multimedia player is invoked when the user selects the 
multimedia annotation indicator 415. 

15 The multimedia player can be, for example, the RealPlayer from 

RealNetworks which supports RealAudio for sound and RealVideo for video. A 
multimedia annotation can also be generated in other formats for audio and 
video and still be supported by the multimedia player provided the appropriate 
plug-ins for these formats are installed. With streaming audio and streaming 

20 video from RealNetworks, the recipients can listen to the audio sound or view 
the video clip while the multimedia annotation is being transferred from its 
location. 

In another embodiment, the recipient may also receive an email message 
that contains the complete multimedia annotation encoded as a Multi-purpose 
25 Internet Mail Extensions (MIME) attachment. The MIME attachment is then 
downloaded into the recipient's system and the multimedia document can then 
be viewed. 
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In one embodiment, the multi-function system can contact the recipients 
by telephone when a multimedia document is sent to the recipient. The multi- 
function system may employ digitized voice and notify the recipients that they 
have documents waiting for them in a multimedia message queue on the server 
5 machine. The recipients can inspect the queue and retrieve the appropriate 

multimedia documents. Alternatively, the recipient may receive no notification. 
In this case, the system may place the multimedia document in the recipient's 
message queue for later inspection. The recipients may retrieve the documents 
in several ways. The users may periodically inspect their message queue on the 

10 server machine and download their most recent messages, much the same way 
voice mail systems operate. 

In one embodiment, the server machine includes a retrieval engine that 
allows the users to search for archived multimedia documents based on their 
contents. In this case, the multimedia documents were created from an image of 

15 the paper documents. The multimedia documents may be subjected to an 
optical character recognition (OCR) process to help with the content-based 
search. In another embodiment, when the multimedia annotation is an audio 
clip, representative audio tracks are generated for each user on the system using 
speech recognition software such as, for example, ViaVoice from IBM of 

20 Armonk, New York. The multimedia documents can be searched based on the 
user's voice as recorded in the audio message. The user's voice can be 
recognized using a speaker or voice identification software such as, for example, 
WaveMakers from Waver Makers Research Inc.. In another embodiment, when 
the multimedia annotation is a video clip, a video track containing an image of 

25 the user can be used as input to face recognition techniques. These various 
embodiments just described allow the recipients to search for the multimedia 
annotations by their textual content as well as by the identity of the user, either 
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by the user's voice or by the user's image. In other embodiments, the 
multimedia documents can be searched using parameters such as, for example, 
the dates of recording, storage, and transmission of the message. 

In another embodiment, the multi-function system of Figure 2 can 
5 operate as a fax machine and captures an image of a fax document while 

sending the fax. In this case, the document printed on the receiver's fax machine 
can contain a printed plain text URL, a bar code representation for the URL, or a 
bar code that represents the entire contents of the message. Similarly, the multi- 
function system of Figure 2 can operate as a scanner. As discussed above, the 

10 console of the multi-function system can be used to specify the recipients of the 
scanned document. Additionally, the console is used to as a means to notify the 
recipients as well as a means to access the server machine. Still in another 
embodiment, the multi-function system of Figure 2 can operate as a printer 
accepting print request from the user's workstation. In this case, the 

15 workstation from which the document is printed is equipped with the necessary 
multimedia input devices to allow the user to generate multimedia annotations. 
When the document is to be printed from the user's workstation, the user added 
the multimedia annotation to the document. A plain text URL or a bar code that 
represents the URL or the complete multimedia annotation can be added by the 

20 user in the same manner previously described. 

Using the method and apparatus described herein provide an individual 
a way for communicating emotions through a paper document. For example. 
Figure 5 shows a user adding a highly emotional message 505 to a document. 
The audio sound makes the user's meaning abundantly clear to the recipient. 

25 Figure 6 illustrates an embodiment of a computer system that can be used 

to perform operations described herein. The various components shown in 
Figure 6 are provided by way of example. Certain components of the computer 
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in Figure 6 can be deleted from the addressing system for a particular 
implementation of the invention. The computer shown in Figure 6 may be any 
type of computer including a general-purpose computer. 

Figure 6 illustrates a system bus 600 to which various components are 
coupled. A processor 602 performs the processing tasks required by the 
computer. Processor 602 may be any type of processing device capable of 
implementing the steps necessary to perform the operations discussed above. 
An input/output (I/O) device 603 is coupled to bus 600 and provides a 
mechanism for communicating with other devices coupled to the computer. For 
example, the user can use the I/O device 603 to specify the name and email 
address of the recipient to whom the multimedia document is to be sent. A 
graphics display adapter 604 is connected to the bus to receive display data 
generated by the processor 602 and store the display data in a display buffer, A 
read-only memory (ROM) 606 and a random access memory (RAM) 608 are 
coupled to bus 600 and provide a storage mechanism for various data and 
information used by the computer, such as, for example, the interface code that 
allow the user to interact with the computer. Although ROM 606 and RAM 608 
are shown coupled to bus 600, in alternate embodiments, ROM 606 and RAM 
608 are coupled directly to processor 602 or coupled to a dedicated memory bus 
(not shown). 

A video console 610 is coupled to the graphics display adapter 604 and 
displays various information and data stored in the display buffer to the user of 
the computer. The data display may include the different multimedia 
annotations stored on the portable storage device. A disk drive 612 is coupled 
to bus 600 and provides a facility for the user to load multimedia annotations 
previously generated on the user's workstation. 
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Figure 7 illustrates an embodiment of a computer-readable medium 700 
containing various sets of instructions, code sequences, configuration 
information, and other data used by a computer or other processing device. The 
embodiment illustrated in Figure 7 is suitable for use with the multimedia 
5 annotation method described above. The various information stored on 

medium 700 is used to perform various data processing operations. Computer- 
readable medium 700 is also referred to as a processor-readable medium. 
Computer-readable medium 700 can be any type of magnetic, optical, or 
electrical storage medium including a diskette, magnetic tape, CD-ROM, 
10 memory device, or other storage medium. 

Computer-readable medium 700 includes interface code 702 that controls 
the flow of information between various devices or components in the computer 
system. Interface code 702 may control the transfer of information within a 
device (e.g., between the processor and a memory device), or between an 
15 input/output port and a storage device. Additionally, interface code 702 may 
control the transfer of information from one device to another. 

Computer-readable medium 700 also includes the multimedia annotation 
generation application 704 that is used to generate the audio and / or video 
messages. Other codes stored on the computer-readable medium 700 may 
20 include the email message generation code 706 to notify the recipient of the 
existence of the multimedia document, a bar code generation code 708 to 
generate a bar code representation of an audio message, and the image 
generation code 712 that generates an image of the paper document. 

From the above description and drawings, it will be understood by those 
25 of ordinary skill in the art that the particular embodiments shown and described 
are for purposes of illustration only and are not intended to limit the scope of 
the invention. Those of ordinary skill in the art will recognize that the invention 
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may be embodied in other specific forms without departing from its spirit or 
essential characteristics. References to details of particular embodiments are not 
intended to limit the scope of the claims. 
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CLAIMS 

What is claimed is: 



1 1. A method comprising: 

2 creating a multimedia annotation for a paper document; and 

3 combining the paper document and the multimedia annotation to form a 

4 first multimedia document. 

1 2. The method of claim 1, wherein the multimedia annotation is represented 

2 as a first bar code printed on the multimedia document. 

1 3. The method of claim 2, wherein the first bar code encodes an audio sound. 

1 4. The method of claim 1, wherein a location indicator associated with the 

2 multimedia annotation is placed on the multimedia document, wherein 

3 the location indicator indicates where the multimedia annotation can be 

4 retrieved and played. 

1 5. The method of claim 4, wherein the location indicator comprises a first 

2 Uniform Resource Locator (URL), and a second bar code, wherein the 

3 first URL is indicated in plain text, and wherein the second bar code 

4 represents the first URL. 
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1 6. The method of claim 4, wherein the muhimedia annotation comprises an 

2 audio sound, a video clip, and a combination of both the audio sound and 

3 the video clip. 

1 7. The method of claim 1, wherein the first multimedia document is a paper 

2 document. 

1 8. The method of claim 1, further comprising: 

2 generating an image of the paper document; 

3 combining the image of the paper document and the multimedia 

4 annotation to form a second multimedia document; and 

5 storing the image of the paper document and the multimedia annotation. 

1 9. The method of claim 8, wherein the second multimedia document is 

2 represented as a second Uniform Resource Locator (URL), and wherein 

3 the image of the paper document and the multimedia annotation is 

4 accessed with the second URL. 

1 10. The method of claim 9, wherein a third bar code is used to represent the 

2 second URL. 

1 11. The method of claim 8, further comprising sending the second multimedia 

2 document to a recipient by electronic mail 
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1 12, The method of claim 11, wherein the recipient receives the image of the 

2 paper document and the multimedia annotation in the form of Multi- 

3 purpose Internet Mail Extension (MIME). 

1 13. A machine-readable medium providing instructions, which when executed 

2 by a set of one or more processors, cause said set of processors to perform 

3 the following: 

4 creating a multimedia annotation for a paper document; and 

5 combining the paper document and the multimedia annotation to form a 

6 first multimedia document. 

1 14. The machine-readable medium of claim 13, wherein the multimedia 

2 annotation is represented as a first bar code printed on the multimedia 

3 document. 

1 15. The machine-readable medium of claim 14, wherein the first bar code 

2 encodes an audio sound. 

1 16. The machine-readable medium of claim 13, wherein a location indicator 

2 associated with the multimedia annotation is placed on the multimedia 

3 document, wherein the location indicator indicates where the multimedia 

4 annotation can be retrieved and played. 

1 17. The machine-readable medium of claim 16, wherein the location indicator 

2 comprises a first Uniform Resource Locator (URL), and a second bar 
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3 code, wherein the first URL is indicated in plain text, and wherein the 

4 second bar code represents the first URL. 

1 18. The machine-readable medium of claim 16, wherein the multimedia 

2 annotation comprises an audio sound, a video clip, and a combination of 

3 both the audio sound and the video clip. 

1 19. The machine-readable medium of claim 13, wherein the first multimedia 

2 document is a paper document. 

1 20. The machine-readable medium of claim 13, further comprising: 

2 generating an image of the paper document; 

3 combining the image of the paper document and the multimedia 

4 annotation to form a second multimedia document; and 

5 storing the image of the paper document and the multimedia annotation. 

1 21. The machine-readable medium of claim 20, wherein the second multimedia 

2 document is represented as a second Uniform Resource Locator (URL), 

3 and wherein the image of the paper document and the multimedia 

4 annotation is accessed with the second URL. 

1 22. The machine-readable medium of claim 21, wherein a third bar code is 

2 used to represent the second URL. 

1 23. The machine-readable medium of claim 20, further comprising sending the 

2 second multimedia document to a recipient by electronic mail. 
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1 24. The machine-readable medium of claim 23, wherein the recipient receives 

2 the image of the paper document and the multimedia annotation in the 

3 form of Multi-purpose Internet Mail Extension (MIME). 

1 25. A computer system, comprising: 

2 a bus; 

3 a data storage device coupled to the bus; and 

4 a processor coupled to the data storage device, the processor operable to 

5 receive instructions which, when executed by the processor, cause the 

6 processor to perform a method comprising: 

7 creating a multimedia annotation for a paper document; and 

8 combining the paper document and the multimedia annotation to 

9 form a first multimedia document. 

1 26. The computer system of claim 25, wherein the multimedia annotation is 

2 represented as a first bar code printed on the multimedia document; 

1 27. The computer system of claim 26, wherein the first bar code encodes an 

2 audio sound. 

1 28. The computer system of claim 25, wherein a location indicator associated 

2 with the multimedia armotation is placed on the multimedia document, 

3 wherein the location indicator indicates where the multimedia annotation 

4 can be retrieved and played. 
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1 29. The computer system of claim 28, wherein the address comprises a first 

2 Uniform Resource Locator (URL), and a second bar code, wherein the 

3 first URL is indicated in plain text, and wherein the second bar code 

4 represents the first URL. 

1 30. The computer system of claim 28, wherein the multimedia annotation 

2 comprises an audio sound, a video clip, and a combination of both the 

3 audio sound and the video clip. 

1 31. The computer system of claim 25, wherein the first multimedia document is 

2 a paper document. 

1 32. The computer system of claim 25, further comprising: 

2 generating an image of the paper document; 

3 combining the image of the paper document and the multimedia 

4 annotation to form a second multimedia document; and 

5 storing the image of the paper document and the multimedia annotation. 

1 33. The computer system of claim 32, wherein the second multimedia 

2 document is represented as a second Uniform Resource Locator (URL), 

3 and wherein the image of the paper document and the multimedia 

4 annotation is accessed with the second URL. 

1 34. The computer system of claim 33, wherein a third bar code is used to 

2 represent the second URL. 
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1 35. The computer system of claim 32, further comprising sending the second 

2 multimedia document to a recipient by electronic mail. 

1 36. The computer system of claim 35, wherein the recipient receives the image 

2 of the paper document and the multimedia annotation in the form of 

3 Multi-purpose Internet Mail Extension (MIME). 

1 37. A method comprising: 

2 creating a document to be used with a multimedia annotation; 

3 creating the multimedia annotation; 

4 storing an image of the document and the multimedia annotation; and 

5 combining the document and the multimedia annotation to form a 

6 multimedia document. 

1 38. The method of claim 37, wherein combining the document and the 

2 multimedia annotation comprises: 

3 creating a paper multimedia document by associating the multimedia 

4 annotation with the paper document; and 

5 creating an electronic multimedia document by associating the multimedia 

6 annotation with the image of the document. 

1 39. The method of claim 38, wherein the multimedia annotation associated 

2 with the paper multimedia document is represented as a first bar code 
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3 printed on the paper multimedia document wherein the first bar code 

4 encodes an audio sound. 

1 40* The method of claim 38, wherein a location indicator of the multimedia 

2 annotation associated with the paper multimedia annotation is placed on 

3 the paper multimedia document, wherein the location indicator indicates 

4 where the multimedia annotation can be retrieved and played, 

1 41. The method of claim 40, wherein the location indicator comprises a 

2 Uniform Resource Locator (URL), and a second bar code, wherein the 

3 URL is indicated in plain text, and wherein the second bar code 

4 represents the URL. 

1 42. The method of claim 40, wherein the multimedia annotation comprises an 

2 audio soumd, a video clip, and a combination of both the audio sound and 

3 the video clip, 

1 43. The method of claim 38, further comprising sending the electronic 

2 multimedia document to a recipient, wherein the recipient receives the 

3 electronic multimedia document in the form of an attachment to an 

4 electronic mail. 
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ABSTRACT OF THE DISCLOSURE 



A method and apparatus for adding multimedia annotations to paper 
documents is disclosed. In one embodiment, a user generates a multimedia 

5 annotation to be combined to a paper document. The multimedia annotation is 
combined with the paper document to form a multimedia document. In another 
embodiment, an image of the paper document is generated. The multimedia 
annotation is combined to the image of the paper document to form an 
electronic multimedia document. The multimedia annotation is attached to the 

10 paper document in the form of plain text URL or bar code representing the URL. 
The electronic multimedia document is stored for later retrieval. 
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, ht1p://www.rsv.ricoh.com/mmdoc 421 .html 



f\OiiK^ 4A 




FIGURE 46 



/ 



Look» this needs to be done today. . .not 
tomorrow., .the meeting can not be 
delayed. . .do whatever it takes. . ,1 expect to see 
it on my desk first thing in the morning! 




\\ or WAN 



an .au + .inpg 



FIGURE ^ 



^3 
m 

m 
m 

y 

w 

m 
□ 
□ 



Processor 
602 


< — ► 


< — ► 


Disk Drive 
612 







Input / Output 

Device 
603 






Read-only Memory 
606 


< — ► 


< — ► 










Random Access Memory 
608 






Display Adapter 
604 


< — ► 


< — ► 



t 

600 



Video Console 
610 



FIGURE 6 



Interface Program 
702 



Multimedia 
Aimotation Generation 
704 



Email Generation 



706 



Bar Code 
Generation 

708 



Image 
Generation 

712 



700 



Q ' 

^ Figure 7 

a 

Ui 

Q 

W 
I- 

m 
a 
a 



Attorney's Docket No.: 74451 .P11 4 Patent 
DECLARATION AND POWER OF ATTORNEY FOR PATENT APPLICATION 



As a below nanned inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below, next to my name. 

I believe I am the original, first, and sole inventor (if only one name is listed below) or an original, 
first, and joint inventor (if plural names are listed below) of the subject matter which is claimed and 
for which a patent is sought on the invention entitled 

"MULTIMEDIA DOCUMENT ANNOTATION" 



the specification of which 

X is attached hereto. 

was filed on as 

United States Application Number ^ 

or PCT International Application Number 

and was amended on 

(if applicable) 

I hereby state that 1 have reviewed and understand the contents of the above-identified 
specification, including the claim(s), as amended by any amendment referred to above. I do not 
know and do not believe that the claimed invention was ever known or used in the United States of 
America before my invention thereof, or patented or described in any printed publication in any 
country before my invention thereof or more than one year prior to this application, that the same 
was not in public use or on sale in the United States of America more than one year prior to this 
application, and that the invention has not been patented or made the subject of an inventor's 
certificate issued before the date of this application in any country foreign to the United States of 
America on an application filed by me or my legal representatives or assigns more than twelve 
months (for a utility patent application) or six months (for a design patent application) prior to this 
application. 

I acknowledge the duty to disclose all information known to me to be material to patentability as 
defined in Title 37, Code of Federal Regulations, Section 1.56. 

I hereby claim foreign priority benefits under Title 35, United States Code, Section 1 19(a)-(d), of any 
foreign appHcation(s) for patent or inventor's certificate listed below and have also identified below 
any foreign application for patent or inventor's certificate having a filing date before that of the 
application on which priority is claimed: 
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Priority 



Prior Foreian ADDlication(s) 






Claimed 


(Number) 


(Country) 


(Day/MonthA'ear Filed) 


Yes No 


(Number) 


(Country) 


(Day/MonthA'ear Filed) 


Yes No 



(Number) (Country) (Day/Month/Year Filed) Yes No 



I hereby claim the benefit under title 35, United States Code, Section 119(e) of any United States 
provisional application(s) listed below: 



(Application Number) Filing Date 



(Application Number) Filing Date 



1 hereby claim the benefit under Title 35, United States Code, Section 120 of any United States 
application(s) listed below and, insofar as the subject matter of each of the claims of this application 
is not disclosed in the prior United States application in the manner provided by the first paragraph 
of Title 35, United States Code, Section 1 12, 1 acknowledge the duty to disclose all information 
known to me to be material to patentability as defined in Title 37, Code of Federal Regulations, 
Section 1 .56 which became available between the filing date of the prior application and the national 
or PCT international filing date of this application: 



(Application Number) Filing Date (Status -- patented, 

pending, abandoned) 



(Application Number) Filing Date (Status - patented, 

pending, abandoned) 

I hereby appoint the persons listed on Appendix A hereto (which is incorporated by reference and a 
part of this document) as my respective patent attorneys and patent agents, with full power of 
substitution and revocation, to prosecute this application and to transact all business in the Patent 
and Trademark Office connected herewith. 

Send correspondence to Michael J. Mallie , BLAKELY, SOKOLOFF, TAYLOR & 

(Name of Attorney or Agent) 
ZAFMAN LLP, 12400 Wilshire Boulevard 7th Floor, Los Angeles, California 90025 and direct 

telephone calls to Michael J, Mallie , (408) 720-8300. 

(Name of Attorney or Agent) 
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I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on Information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so made 
are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United 
States Code and that such willful false statements may jeopardize the validity of the 
application or any patent issued thereon. 

Full Name of Sole/First Inventor Jonathan J. Hull 



Inventor's Signature Date . 

Residence Citizenship . 



(City, State) (Country) 
Post Office Address 

Full Name of Second/Joint Inventor Michael Baxter ^ 



Inventor's Signature „ Date . 

Residence Citizenship 



(City, State) (Country) 
Post Office Address 



Full Name of Third/Joint Inventor . 



Inventor's Signature Date . 

Residence Citizenship . 



(City, State) (Country) 
Post Office Address „ 

Full Name of Fourth/Joint Inventor 



Inventor's Signature Date . 

Residence „ Citizenship . 



(City, State) (Country) 
Post Office Address 
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APPENDIX A 



William E. Alford, Reg. No. 37,764; Farzad E. Amini, Reg. No. P42,261; Aloysius T. C. AuYeung, Reg. No. 
35,432; William Thomas Babbitt, Reg. No. 39,591; Carol F. Barry, Reg. No. 41,600; Jordan Michael 
Becker, Reg. No. 39,602; Bradley J. Bereznak, Reg. No. 33,474; Michael A. Bernadicou, Reg. No. 35,934; 
Roger W. Blakely, Jr., Reg. No. 25,831; Gregory D. Caldwell, Reg. No. 39,926; Ronald C. Card, Reg. No. 
44,587; Andrew C. Chen, Reg. No. 43,544; Thomas M. Coester, Reg. No. 39,637; Alin Corie, Reg. No. 
P46,244; Dennis M. deGuzman, Reg. No. 41,702; Stephen M. De Klerk, under 37 C.RR. § 10.9(b); 
Michael Anthony DeSanctis, Reg. No. 39,957; Daniel M. De Vos, Reg. No. 37,813; Robert Andrew Diehl, 
Reg. No. 40,992; Sanjeet Dutta, Reg. No. P46,145; Matthew C. Pagan, Reg. No. 37,542; Tarek N. Fahmi, 
Reg. No. 41,402; Paramita Ghosh, Reg. No. 42,806; James Y. Go, Reg. No. 40,621; James A. Henry, 
Reg. No. 41,064; Willmore F. Holbrow III, Reg. No. P41,845; Sheryl Sue Holloway, Reg. No. 37,850; 
George W Hoover II, Reg. No. 32,992; Eric S. Hyman, Reg. No. 30,139; William W. Kidd, Reg, No. 
31,772; Sang Hui Kim, Reg. No. 40,450; Eric T. King, Reg. No. 44,188; Erica W. Kuo, Reg. No. 42,775; 
Kurt P. Leyendecker, Reg. No. 42,799; Michael J. Mallie, Reg. No. 36,591; Andre L. Marais, under 37 
C.F.R. § 10.9(b); Paul A. Mendonsa, Reg. No. 42,879; Darren J. Milliken, Reg. 42,004; Lisa A. Norris, 
Reg. No. 44,976; Chun M. Ng, Reg. No. 36,878; Thien T. Nguyen, Reg. No. 43,835; Thinh V. Nguyen, 
Reg. No. 42,034; Dennis A. Nicholls, Reg. No. 42,036; Daniel E. Ovanezian, Reg. No. 41 ,236; Marina 
Portnova, Reg. No. P45,750; Babak Redjaian, Reg. No. 42,096; William F. Ryann, Reg. 44,313; James 
H. Salter, Reg. No. 35,668; William W. Schaal, Reg. No. 39,018; James C. Scheller, Reg. No. 31,195; 
Jeffrey Sam Smith, Reg. No. 39,377; Maria McCormack Sobrino, Reg. No. 31,639; Stanley W. Sokoloff, 
Reg. No. 25,128; Judith A. Szepesi, Reg. No. 39,393; Vincent P. Tassinari, Reg. No. 42,179; Edwin H. 
Taylor, Reg. No. 25,129; John F. Travis, Reg. No. 43,203; George G. C. Tseng, Reg. No. 41 ,355; Joseph 
A. Twarowski, Reg. No. 42,191; Lester J. Vincent, Reg. No. 31,460; Glenn E. Von Tersch, Reg. No. 
41,364; John Patrick Ward, Reg. No. 40,216; Mark L. Watson. Reg. No. P46,322; Thomas C. Webster, 
Reg. No. P46,154; Charles T. J. Weigell, Reg. No. 43,398; Kirk D. Williams, Reg. No. 42,229; James M. 
Wu, Reg. No. 45,241; Steven D. Yates, Reg. No. 42,242; and Norman Zafman, Reg. No. 26,250; my 
patent attorneys, and Justin M. Dillon, Reg. No. 42,486; my patent agent, of BLAKELY, SOKOLOFF, 
TAYLOR & ZAFMAN LLP, with offices located at 12400 Wilshire Boulevard, 7th Floor, Los Angeles, 
California 90025, telephone (310) 207-3800, and James R. Thein, Reg. No. 31 ,710, my patent attorney. 



Rev. 02/07/00 (D1) 



-4- 



APPENDIX B 



Title 37, Code of Federal Regulations, Section 1.56 
Duty to Disclose Information Material to Patentability 

(a) A patent by its yery nature is affected with a public interest. The public interest is best served, 
and the most effective patent examination occurs when, at the time an application is being examined, the 
Office is aware of and evaluates the teachings of all information material to patentability. Each individual 
associated with the filing and prosecution of a patent application has a duty of candor and good faith in 
dealing with the Office, which includes a duty to disclose to the Office all information known to that individual 
to be materia! to patentability as defined in this section. The duty to disclosure information exists with respect 
to each pending claim until the claim is cancelled or withdrawn from consideration, or the application becomes 
abandoned. Information material to the patentability of a claim that is cancelled or withdrawn from 
consideration need not be submitted if the information is not material to the patentability of any claim 
remaining under consideration in the application. There is no duty to submit information which is not material 
to the patentability of any existing claim. The duty to disclosure all information known to be material to 
patentability is deemed to be satisfied if all information known to be materia! to patentability of any claim 
issued in a patent was cited by the Office or submitted to the Office in the manner prescribed by §§1 .97(b)-(d) 
and 1 .98. However, no patent will be granted on an application in connection with which fraud on the Office 
was practiced or attempted or the duty of disclosure was violated through bad faith or intentional misconduct. 
The Office encourages applicants to carefully examine: 

(1) Prior art cited in search reports of a foreign patent office in a counterpart application, and 

(2) The closest information over which individuals associated with the filing or prosecution of a 
patent application believe any pending claim patentably defines, to make sure that any material information 
contained therein is disclosed to the Office. 

(b) Under this section, information is material to patentability when it is not cumulative to 
information already of record or being made or record in the application, and 

(1) It establishes, by itself or in combination with other information, a prima facie case of 
unpatentability of a claim; or 

(2) It refutes, or is inconsistent with, a position the applicant takes in: 

(i) Opposing an argument of unpatentability relied on by the Office, or 

(ii) Asserting an argument of patentability. 

A prima facie case of unpatentability is established when the information compels a conclusion that a claim is 
unpatentable under the preponderance of evidence, burden-of-proof standard, giving each term in the claim 
its broadest reasonable construction consistent with the specification, and before any consideration is given to 
evidence which may be submitted in an attempt to establish a contrary conclusion of patentability. 

(c) Individuals associated with the filing or prosecution of a patent application within the 
meaning of this section are: 

(1) Each inventor named in the application; 

(2) Each attorney or agent who prepares or prosecutes the application; and 

(3) Every other person who is substantively involved in the preparation or prosecution of the 
application and who is associated with the inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application. 

(d) Individuals other than the attorney, agent or inventor may comply with this section by 
disclosing information to the attorney, agent, or inventor. 
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